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(54) Information processing apparatus, method, and computer-readable medium 



(57) A system and method for creating multimedia 
information synchronized with a video signal (e.g., a pro- 
gram of a television broadcast) are provided. The mul- 
timedia information may be formatted in accordance 
with, for example, the MHEG standard or the ATVEF 
standard. The system includes means for specifying 
and verifying the contents of, and synchronizations for, 
the multimedia information. A scenario creation pro- 



gram creates a scenario composed of a plurality of 
scenes synchronized with playback of the video. A 
scene creation program combines objects consisting of 
text, images, audio, operation buttons and the like to 
create scenes (pages). A contents management pro- 
gram, a scenario management program and a scene 
management program set a directory of the contents, 
the scenarios and the scenes. 
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Description 

[0001 ] The present invention relates to an inf ornnation 
processing apparatus, method and computer-readable 
medium. 

[0002] Standards exist, or have been proposed, for 
displaying information and data, including moving imag- 
es, still images, sound and text in response to a user's 
interactive commands, and for synchronously display- 
ing such information and data. One such standard, pro- 
posed by the International Organization for Standardi- 
zation (ISO), is that defined by the Multimedia and Hy- 
permedia Information Coding Experts Group (l\/!HEG). 
Another standard is that defined by the Advanced Tele- 
vision Enhancement Forum (ATVEF), a consortium of 
broadcast and cable networks, consumer electronics 
companies, television transport operators and technol- 
ogy companies, 

[0003] Information such as that corresponding to the 
MHEG standard or the ATVEF standard (hereinafter re- 
ferred to generally as "multimedia information") can be 
provided to a viewer in a television broadcast. Other 
methods of transmission and display are contemplated, 
however, including over the Internet to computers. 
[0004] Multimedia information generally changes little 
with respect to the video to which it corresponds. To pro- 
vide multimedia information at an appropriate time, how- 
ever, the multimedia information should be synchro- 
nized with the video to which it corresponds (for exam- 
ple, the moving images of a television broadcast). An 
authoring system or authoring tool for creating such syn- 
chronized multimedia information, however, does not 
exist. An object of the present invention is to provide 
such an authoring system or authoring tool. 
[0005] Various respective aspects and features of the 
invention are defined in the appended claims. 
[0006] Embodiments of the invention can provide an 
information processing system and method. The system 
includes means for specifying multimedia information 
and video with which the multimedia information is syn- 
chronized, means for creating the multimedia informa- 
tion and means for setting the timing for displaying 
scenes of the multimedia information to synchronize this 
information with the video. The method includes speci- 
fying multimedia information and video with which the 
multimedia information is to be synchronized, creating 
the multimedia information and setting timing for dis- 
playing scenes of the multimedia information to syn- 
chronize this information with the video. A computer- 
readable medium containing code for executing this 
method also is provided. In preferred embodiments, the 
multimedia information is either MHEG information or 
ATVEF information. 

[0007] The specifying means preferably comprises a 
contents creating section for identifying the contents of 
the multimedia information and of the video with which 
the multimedia information is to be synchronized. The 
creating means preferably comprises a scene creating 



section for creating scenes of the multimedia informa- 
tion, and the setting means preferably comprises means 
for identifying the video's starting time for displaying 
scenes of the multimedia information synchronized with 

5 such starting time. 

[0008] The infornnation processing system preferably 
also corriprises a display, and a display controller, for 
displaying a time axis of the video and symbols corre- 
sponding to a hierarchy of the scenes. The information 

10 processing system preferably also comprises means for 
setting a synchronization flag for activating or deactivat- 
ing synchronized playback of the multimedia informa- 
tion with the video. A mid-processing playback flag for 
allowing or disallowing mid-processing display of an im- 

75 age also preferably is provided. Disallowing such mid- 
processing display prevents the image's display to a 
viewer who begins viewing the video at a point when the 
image's display would be meaningless. 
[0009] The information processing system preferably 

20 also comprises managing means for organizing, in a hi- 
erarchical manner, the multimedia information into con- 
tents, scenarios and scenes and for managing this hier- 
archical structure. Means for setting the priority level of 
a scene, such that the rate at which the scene is trans- 

25 mitted from a broadcast station corresponds to a view- 
er's likelihood of selecting the scene for display, also 
preferably Is provided. 

[0010] The information processing apparatus prefer- 
ably also comprises means for setting a display mode 

30 flag for determining whether a viewer can control the 
multimedia information's display. The setting of this flag 
prevents the viewer from switching off this display. 
[0011] The invention will now be described by way of 
example with reference to the accompanying drawings, 

35 throughout which like parts are referred to by like refer- 
ences, and in which: 

Fig. 1 is a block diagram of a system for broadcast- 
ing multimedia information created using the appa- 
40 ratus and method of an embodiment of the present 
invention; 

Fig. 2 is a block diagram of an apparatus for creating 
multimedia information. 

Fig. 3 is a block diagram of the sections of a com- 
45 puter program executed by the multimedia creating 
apparatus of Fig. 2. 

Fig. 4 is an illustration of the objects of a scene. 
Fig. 5 is an illustration of a directory for the multi- 
media information showing the information's hierar- 

50 chical structure. 

Fig. 6 is a block diagram of IRD 4 of Fig. 1 . 
Fig. 7 is a block diagram of the sections of a com- 
puter program executed by IRD 4 for playing back 
multimedia information. 

55 Fig. 8 is a flowchart illustrating the steps for confirm- 
ing the synchronization settings created by the mul- 
timedia creating apparatus of Fig. 2. 
Fig. 9 is an example of a display appearing on dis- 
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play apparatus 15 of the multimedia creating appa- 
ratus of Fig. 2 pursuant to steps S1 and S3 of Fig. 8. 
Fig. 10 is a flowchart illustrating the steps for con- 
firming the interactive setting process created by 
the multimedia creating apparatus of Fig. 2. 
Fig. 11 is an illustration of the link structure of 
scenes. 

Fig. 1 2 is an example of a display appearing on dis- 
play apparatus 1 5 in accordance with the process 
of Fig. 10 for confirming interactive settings. 
Fig. 13 is an illustration of the effect of the setting 
of synchronization flags. 

Fig. 14 is an illustration of the effect of the setting 

of mid-processing display flags. 

Fig. 15 illustrates a sending rate tor scenes. 

Fig. 16 further illustrates a sending rate for scenes. 

[0012] A broadcasting system for broadcasting video 
and multimedia information, such as, e.g., MHEG infor- 
mation or ATVEF information, is shown in Fig. 1 . Broad- 
cast station 1 transmits television broadcasting signals 
containing multiplexed multimedia information signals. 
Antenna 3 receives these signals via satellite 2. The re- 
ceived signals are supplied to IRD (Integrated Receiver 
and Decoder) 4 which extracts the television and multi- 
media signals on a predetermined channel. The extract- 
ed signals are supplied to display apparatus 5 for dis- 
playing the multimedia information superposed upon 
the video of a television broadcast, 
[0013] Fig. 2 is a block diagram of an apparatus in 
accordance with an embodiment of the present inven- 
tion for creating multimedia information for transmission 
by broadcast station 1 . The apparatus of Fig. 2 can be 
used to create any form of multimedia information, for 
example, MHEG information or ATVEF information. For 
the purpose of this example, however, it will be assumed 
that the apparatus of Fig. 2 is used to create MHEG in- 
formation. 

[0014] The apparatus of Fig. 2 includes CPU 11 for 
providing overall control in accordance with a program 
stored in ROM 12 and RAM 13. CPU 11 maybe, e.g., a 
microprocessor or similar device. CPU 11 also executes 
a multimedia authoring program stored on. e.g., hard 
disk 14 for creating MHEG informatbn. The authoring 
program can, of course, be stored on any computer- 
readable medium accessible to CPU 11, e.g,, a floppy 
disk, magnetic tape, optical disk, etc. 
[0015] Display apparatus 15 displays the MHEG in- 
formation, and the video to be synchronized with the 
MHEG information, to an operator. Keyboard 16 and 
mouse 1 7 enable the operator to control the apparatus' 
operations, A communication section 18 including, for 
example, a modem, controls communications over a tel- 
ephone line, the Internet, or the like. Sound playback 
apparatus 1 9 plays back sound to be synchronized with 
the video or MHEG information. 
[0016] Fig. 3 shows the functional blocks (sections) of 
the multimedia information authoring program executed 



by CPU 11. As indicated above, in this example, it is 
assumed that the authoring program is for authoring 
MHEG infornnation. Although each of these bkx;ks re- 
ceives information from one or more of the other blocks 

s and provides information to one or more of the other 
blocks, for clarity of presentation, arrows among the 
blocks indicating such inputs and outputs are omitted. 
[0017] A contents creation section 21 creates con- 
tents information for the MHEG information. This con- 

10 tents information corresponds to a video {e.g., a pro- 
gram of a televiston broadcast) to be synchronized with 
the MHEG information. The contents information is pro- 
vided to the contents management section 24. 
[0018] The contents information corresponds to a 

15 group of scenarios and comprises eight items of infor- 
mation: 

(1 ) contents_name 

(2) service_name 
20 (3) program_name 

(4) es_number 

(5) start_time 

(6) duration 

(7) additionaljnfo 

25 (8) application_number 

[0019] In the contents_name, 

■contents_1988_4_1_1,'' for. example, is recorded to 
identify the name of the contents. In the service_name, 

30 "NHK," for example, is recorded to identify the channel 
on which the program including the multimedia informa- 
tion is broadcast. In the program_name, "baseball,* for 
example, Is recorded to identify the program's name. In 
the es_n umber. "10," for example, is recorded to identify 

35 the number of elementary streams. In the start_time, 
■+00/00/00, ■ for example, is recorded to identify the rel- 
ative time tor starting the display of the MHEG informa- 
tion with respect to the video with which the MHEG in- 
formation Is synchronized. In the duration, '00/00/00," 

^0 for example, is recorded to indicate the duration of the 
multimedia information's display from the relative start 
time. In the additionaljnfo, "scenario 1 for example, is 
recorded to indicate that the information corresponds to 
the first scenario. In the application_number, "3," for ex- 

45 ample, is recorded to indicate the number of scenarios 
contained in the contents. 

[0020] The contents creation section 21 also manag- 
es an application information file for recording the start 
time and duration for broadcasting a scene (described 

so later with reference to Fig. 9). 

[0021] The scenario creation section 22 creates a 
scenario, and scenario information, and provides its out- 
put to the scenario management section 25. A scenario 
is composed of a plurality of scenes synchronized with 

55 the video. The scenario information comprises four 
items of information: 

(1 ) es_name 
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(2) data_type 

(3) custom ized_info 

(4) scene_number 

[0022] In the es_number. 'DATA 1," for example, is 
recorded to identify the name of the elementary stream 
corresponding to the scenario. In the data_type. "mheg. 
" for example, is recorded to identify the type of multi- 
media information. In the customizedjnfo, any specific 
information corresponding to the scenario may be re- 
corded. In the scene_number, "3." for example, is re- 
corded to indicate the number of scenes in the scenario. 
[0023] The scene creation section 23 combines "ob- 
jects." As shown in Fig. 4, the objects make up a scene 
(page) and may be textual information, image informa- 
tion, audio information, operational buttons, and the like. 
The output of scene creation section 23 Is provided to 
scene management section 26. 

[0024] A scene can be switched on and off either by 
synchronization signals contained in the video signal or 
through a viewer's interactive operations, for example, 
activating operational buttons displayed in the scene. 
Further, it is possible to set, in the operational buttons, 
functions for executing predetermined processing, for 
example, to provide textual information to be displayed 
for applying for the purchase of a product. 
[0025] A scene is composed of scene data and six 
items of Information corresponding to these data: 

(1) scene_priority_level 

(2) file_name 

(3) stream_event_name 

(4) stream_event_time 

(5) stream_name 

(6) es_name 

[0026] In the scene_priority_level, '1 for example, is 
recorded to identify the sending rate (described later) 
for the scene. In the file_name, "bitmap .jpg," for exam- 
ple, is recorded to identify the name of an object. In the 
stream_event_name. "stream event 1." for example, is 
recorded to identify the name of the stream event. In the 
stream_event_tlme, ■00/00/00," for example, is record- 
ed to identify the starting time for sending the scene. In 
the stream_name, "stream link 1," for example, is re- 
corded to identify the name of the stream. In the 
es_name, "DATA 1 ", for example, is recorded to identify 
the name of the elementary stream of a link destination. 
[0027] The scene creation section 23 also manages 
scene information and records within this information 
synchronization flags (described below with reference 
to Fig. 13), mid-processing playback mode flags (de- 
scribed below with reference to Fig. 14) and display 
mode flags (described below). 

[0028] The contents management section 24. the 
scenario management section 25 and the scene man- 
agement section 26 set the directory for the contents, 
the scenarios and the scenes. An example is shown In 



Fig. 5. In this example, a contents CI is set in the high- 
est-order directory, and scenarios. SC1. SC2 and SC3 
are set in the next highest-order directory. In the scenar- 
io SC1, three scenes, scene 1, scene 2 and scene 3, 
5 are set. In the scenarios SC2 and SC3, scene 4 and 
scene 5, respectively, are set. 

[0029] Referring again to Fig. 3, a time control section 
27 reads time information for the video to be played back 
by video playback section 29 and outputs this informa- 
10 tion to a video and MHEG display control section 31 . An 
interactive control section 28 records settings for switch- 
ing the display of the scenes in response to operations 
of the user. 

[0030] The video playback section 29 plays back the 
T5 video with which the MHEG information is to be synchro- 
nized. The MHEG playback section 30 plays back the 
MHEG information to verify that the creation, synchro- 
nizations, etc. are correct. The video and MHEG display 
control section 31 controls the display apparatus 15 
20 (Fig. 2) so that the MHEG Infomnation played back by 
the MHEG playback section 30 is displayed superposed 
over the video information played back by the video 
playback section 29. 

[0031] Fig. 6 shows the construction of IRD 4 (Fig. 1 ) 
25 for playing back broadcasted video and multimedia in- 
formation to a viewer. A receiving processing section 41 
containing a down converter, a tuner, a demodulator, a 
descrambler and the like, extracts predetermined chan- 
nel components (television broadcasting signals and 
30 MHEG information signals) from the broadcasted sig- 
nals received on antenna 3 and supplies these compo- 
nents to CPU 42. 

[0032] CPU 42 controls IRD 4 through execution of a 
control program, stored in ROM 43 and RAM 44, and a 

35 MHEG information playback program stored on hard 
disk 45. The broadcasted video and MHEG information 
are displayed on display apparatus 5 (Fig. 1 ). 
[0033] A display control section 46 controls superpos- 
ing of the MHEG information over the broadcasted video 

40 displayed on display apparatus 5. Keyboard 47, mouse 
48, and remote controller 49 provide input from a viewer. 
A communication section 50, for example, a modem, 
controls information communications over a telephone 
line, the Internet and the like. 

45 [0034] Fig. 7 shows the functional blocks of the MHEG 
information playback program for IRD 4 stored on hard 
disk 45 and executed by CPU 42. A video playback sec- 
tion 54 plays back the video of a channel selected by 
the viewer. A MHEG playback section 55 plays back 

50 MHEG information corresponding to this video. A video 
display control section 56 controls the display apparatus 
5 (Fig. 1). This control causes the MHEG information 
from MHEG playback section 55 to be superposed over 
the video from video playback section 54. 

55 [0035] A directory management section 51 manages 
the MHEG Information's directory and provides this in- 
formation to the MHEG playback section 55 in accord- 
ance with predetermined timing. A timing control section 
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52 performs time measurement using, as a reference, 
the time at which the video playback section 54 starts 
playback of the video corresponding to the MHEG infor- 
mation and informs the MHEG playback section 55 of, 
for example, the start_time contained in the contents in- 
formation. 

[0036] An interactive control section 53 detects oper- 
ations by a viewer upon the keyboard 47, the mouse 48 
or the remote controller 49, and performs predetermined 
processing pursuant to these operations, such as 
switching a scene. 

[0037] Referring again to the apparatus for creating 
murtimedia information shown in Fig. 2, by operating the 
keyboard 16 (Fig. 2) and setting predetermined values 
in the contents information (e.g., start_time and dura- 
tion), a scene can be set to be interactively synchronized 
with the video defined in program_name. These settings 
then can be confirmed by CPU 11 in response to an op- 
erator's instructions received from keyboard 16 or 
mouse 17. In response, CPU 11 performs the process- 
ing shown in the flowchart of Fig. 8. 
[0038] As shown in this Figure, in step S1, the time 
control section 27 (Fig. 3) reads the time information 
(start time and end time) of the video to be played back 
by video playback section 29. This time information is 
provided to the video and MHEG display control section 
31 . As shown in Fig. 9, the video and MHEG display 
control section 31 causes display apparatus 15 to dis- 
play a time axis for the video based upon the input time 
information. 

[0039] In step S2, the time control section 27 (Fig. 3) 
reads the playback start time (start_time) and the play- 
back duration (duration) for a scenario to be synchro- 
nized with the video. This information also is provided 
to video and MHEG display control section 31 . The op- 
erator also provides the sending start time (time that 
broadcasting begins) and the sending duration (duration 
of broadcasting) for the MHEG data. This information is 
recorded in the application information file of the con- 
tents creation sectbn 21. The time control section 27 
also reads this infornnation and provides it to the video 
and MHEG display control section 31 . 
[0040] In step S3, the video and MHEG display control 
section 31 causes display apparatus 1 5 to display the 
playback start time, the playback duration, the sending 
start time and the sending duration along the time axis 
displayed in step SI. This display, shown in Fig. 9, en- 
ables the operator to visualize, and confirm, the syn- 
chronization settings for the MHEG information. 
[0041] In the example shown in Fig. 9, the playback 
start times, the playback durations, the sending start 
times and the sending durations for the scenarios SCI 
and SC2 are set with respect to the start lime for the 
corresponding video program. For scenario SCI . in this 
example, the playback start time is set for the same time 
as the program start time, the playback duration is set 
for 1 0 minutes, the sending start time is set for 5 minutes 
before the program start time and the sending duration 



is set for 20 minutes. For scenario SC2, the playback 
start time is set for 20 minutes after the program start 
time, the playback duration is set for 10 minutes, the 
sending start time is set for 1 5 minutes after the program 

s start time, and the sending duration is set for 20 minutes. 
[0042] During the 5 minute durations after completion 
of playback of the scenarios, a viewer can, for example, 
make a telephone call to purchase an item that was dis- 
played (in the case of a shopping program). Since the 

10 MHEG creating apparatus enables recording of the set- 
tings such that the MHEG information is transmitted only 
during the necessary times, the transmissk>n channel is 
used effectively for transmitting a plurality of MHEG data 
in a time divisional manner. 

IS [0043] The process of confirming the interactive set- 
tings for the MHEG information prepared by the creation 
apparatus of Fig. 2 is described with reference to Figs. 
10, 11 and 12. This process enables the user to visual- 
ize, and confirm the accuracy of, the scenes' interactive 

20 relationships with respect to time (link relationships) af- 
ter the synchronization settings, such as playback start 
time and the like, have been set and the scenes within 
the scenario have been linked. As an example, as 
shown in Fig. 11, assume that scenes 2 through 5 are 

25 set as possible selections for display by the viewer after 
scene 1 and that scenes 6 through 8 are set as possible 
selections for display by the viewer after scene 2. 
[0044] Upon receipt of an instruction for starting the 
confirmation process on keyboard 16 or mouse 17 (Fig. 

30 2), the time control section 27 (Fig. 3) reads, in step S21 
(Fig. 10), the time information (start time and end time) 
for the video to be played back by the video playback 
section 29 (Fig. 3) and supplies this information to the 
video and MHEG display control section 31 (Fig. 3). As 

35 shown in Fig. 1 2, the video and MHEG display control 
section 31 causes display apparatus 15 (Fig. 2) to dis- 
play a time axis for the video based upon the received 
time information. 

[0045] In step S22, the time control section 27 next 

40 reads the playback start time and the playback du ration 
for the first scene (scene 1 ) and supplies this information 
to the video and MHEG display control section 31. In 
step S23, as shown in Fig. 1 2, the video and MHEG dis- 
play control section 31 causes display apparatus 15 to 

45 display, below the time axis for the video, a symbol in- 
dicating the playback duration for scene 1. This symbol 
starts 10 minutes after the video start time. The video 
and MHEG display control section 31 also causes dis- 
play apparatus 1 5 to display a symbol for scene 2 adja- 

50 cent to the symbol for scene 1 , The video and MHEG 
display control section 31 further causes display appa- 
ratus 15 to display a triangular mark within the symbol 
for scene 2 to indicate that other scenes are present 
(possible other selections for display after scene 1 ) in 

55 the same hierarchy as that for scene 2. 

[0046] In step S24, the interactive control section 28 
determines whether the user has selected the symbol 
for scene 2 (e.g. , clicked by the mouse 1 7). If this section 
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determines that the symbol for scene 2 has been select- 
ed, the process proceeds to step S25. 
[0047] In step S25. the video and MHEG display con- 
trol section 31 pulls down the symbols for scenes 3 
through 5, as shown in Fig. 12. and causes display ap- 5 
paratus 1 5 to display these symbols below the symbol 
for scene 2. 

[0048] In step S26, the interactive control section 28 
determines whether the user has selected one of the 
symbols for scenes 2 through 5. If this section deter- 
mines that one of these symbols has been selected, the 
process proceeds to step S27. In this example, it Is as- 
sumed that the symbol for scene 2 has been selected. 
[0049] In step S27, as shown in Fig. 12, the video and 
MHEG display control section 31 causes the display ap- 
paratus 15 to display the symbol for scene 2 below the 
symbol for scene 1 and further display the symbol for 
scene 6, which is linked to scene 2, in a symbol adjacent 
to the symbol for scene 2. Within the symbol for scene 
6. a triangular mark is displayed to indicate that other 
scenes are present (possible other selections for display 
after scene 2) in the same hierarchy as that for scene 6. 
[0050] In step S28, the interactive control section 28 
determines whether the symbol for scene 6 has been 
selected. If this section determines that the symbol for 
scene 6 has been selected, the process proceeds to 
step S29 where the video and MHEG display control 
section 31 pulls down the symbols for scenes 7 and 8 
and displays these symbols below the symbol for scene 
6. 

[0051] In step S30. the interactive control section 28 
determines whether one of the symbols for scenes 6 
through 8 has been selected. If this section determines 
that one of these symbols has been selected, the proc- 
ess proceeds to step S31 . In this example, it is assumed 
that the symbol for scene 6 has been selected. 
[0052] In step S31 , the video and MHEG display con- 
trol section 31 displays the symbol for scene 6 below 
that for scene 2. Since no scenes are present in a hier- 
archy below that for scene 6, the processing terminates. 
[0053] The contents creation section 21 (Fig. 3) also 
enables a user to set synchronization signals (flags) 
within the stream of video information. These flags are 
set in the scene information file by the scene creation 
section 23 and are processed by IRD 4 (Fig. 1 ). The set- 
ting of a synchronization flag for a scene to the on state 
causes the scene's switching to be synchronized with 
the video. The setting of such a flag to the off state dis- 
ables such synchronization with the video. 
[0054] For example, as shown in Fig. 1 3, assume that 
scenes 1 through 4 form scenario SCI and a synchro- 
nization flag for switching to scene 3 in the middle of 
scene 2 is received from the video signal. Assume fur- 
ther that in pattern 1 , the synchronization flag is off and 
in pattern 2, the synchronization flag is on. As shown in 
Fig. 13, in pattern 1 (synchronization flag off), after 
scene 2 is terminated, scene 3 is played back (the play- 
back is not synchronized with the video). In pattern 2 



(synchronization flag on), although playback of scene 2 
is incomplete, the scene nevertheless is switched to 
scene 3 (synchronized playback is performed). 
[0055] The scene creation section 23 also may set a 
mid-processing playback mode flag in the scene infor- 
mation file. For example, as shown in Fig. 14, assume 
that scenes 1 through 3 fomri scenario SCI , that these 
scenes are set to be played back in sequence and that 
a viewer's playback of the video signals begins during 
mid-processing of scene 2. Assume further that, in pat- 
tern 3, the mid-processing playback mode flag is set to 
the on state and in pattern 4, the mid-processing play- 
back mode flag is set to the off state. As shown in Fig. 

14. in pattern 3, the playback of scene 2 is started during 
the mid-processing of this scene (since the mid- 
processing playback mode flag is on, a mid-processing 
display occurs). In pattern 4, on the other hand, when 
playback of the video signals begins (for example, IRD 
4 is turned on and starts to decode received data), the 
play back of scene 2 does not begin even if scene data 
are transmitted (since the mid-processing playback 
mode flag is off, a mid-processing display does not oc- 
cur). The first scene played back is scene 3 which be- 
gins after termination of the time for playback of scene 
2 and at the time set for starting play back of scene 3. 
This flag prevents the display of multimedia information 
which is meaningful only if displayed in synchronization 
with the video. 

[0056] The scene creation section 23 also can set a 
display mode flag for transmission to IRD 4 to disable a 
viewer's ability to turn off multimedia information other- 
wise selectable by the viewer. For example, this flag 
may be used if the scene is a commercial synchronized 
with a commercial video and display of the multimedia 
information on display apparatus 5 is desired regardless 
of the viewer's intent. If the display mode flag is set to 
the on state, the scene is displayed regardless of the 
viewer's selection on the controls for IRD 4. If the display 
mode flag is set to the off state, the viewer's selection 
on the controls for IRD 4 determines whether the MHEG 
information is displayed. 

[0057] The scene creation section 23 also can set. in 
the scene_priorityJevel for the scene, a number to de- 
termine the rate at which the scene is transmitted from 
broadcasting system 1 . For example, as shown in Fig. 

15, if scenes 1 through 3 form scenark> SCI and scene 
1 (the initial scene) is selected for display most frequent- 
ly by viewers, the sending rate for scene 1 can be set 
to, e.g., "2" and the sending rate for scenes 2 and 3 can 
be set to, e.g., "1 As shown in Fig. 1 6, in the information 
stream for scenario SCI whose transmission rate is set 
in this manner, the information for scene 1 is sent twice 
as frequently as that for scenes 2 and 3. This transmis- 
sion rate decreases the time before the most frequently 
viewed scene (scene 1 ) is displayed to the viewer upon 
selection. Although a ratio of integers is illustrated for 
the sending rate, this setting could be any ratio. 
[0058] Although the synchronization settings for the 
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MHEG information with respect to the video are de- 
scribed above as being set using relative time, these 
synchronization settings also could be set with respect 
to real or absolute time. 

[0059] A computer program for performing the above- 
described methods may be stored on any computer- 
readable medium, such as an optical disk, magnetic 
tape or CD-ROM, and such a program may be distrib- 
uted through any means of distribution such as the In- 
ternet or digital satellites. 

[0060] In summary, embodiments of the invention can 
provide an information processing apparatus, method 
and computer-readable medium for creating, displaying 
and synchronizing audio, pictorial and textual data with 
video information. 



Claims 

1. An information processing apparatus for creating 
multimedia information to accompany video infor- 
mation transmitted for display to a viewer, said in- 
formation processing apparatus comprising: 

creating means for creating said multimedia in- 
formation; 

identifying means for identifying video informa- 
tion with which said multimedia information is 
to be synchronized; and 
setting means for setting the timing for display- 
ing one or more scenes of said multimedia in- 
formation so as to cause said displaying to be 
synchronized with the display of said video in- 
formation. 

2. An information processing apparatus as in claim 1 , 
wherein said multimedia information conforms to 
the MHEG standard. 

3. An information processing apparatus as in claim 1 . 
wherein said multimedia information conforms to 
the ATVEF standard. 

4. An information processing apparatus as in claim 1 , 
wherein said setting means comprises means tor 
setting said timing with respect to the relative time 
for the display of said video information. 

5. An information processing apparatus as in claim 1 , 
further comprising: 

time-axis display means for displaying an axis 
corresponding to the time for the display of said 
video information; and 

display control means for displaying, relative to 
said axis, symbols corresponding to the timing 
for displaying said one or more scenes. 



6. An information processing apparatus as in claim 1 , 
further comprising deactivation means for deacti- 
vating the synchronized display, with said video in- 
formation, of a scene of said multimedia informa- 

5 tion. 

7. An information processing apparatus as in claim 1 , 
further comprising mid-processing means for set- 
ting a flag to prevent the display of a scene of said 

10 multimedia information whose playback is synchro- 
nized to begin, with respect to said video informa- 
tion, at a point in said video information which oc- 
curs before the point at which said viewer begins 
playing back said video information. 

IS 

8. An information processing apparatus as in claim 1 , 
further comprising management means for organ- 
izing and managing said multimedia information in 
a hierarchy of contents, scenarios and scenes. 

20 

9. An information processing apparatus as in claim 1 , 
further comprising transmission rate setting means 
for setting the rate at which a plurality of said scenes 
are transmitted to said viewer. 

25 

10. An information processing apparatus as in claim 1. 
further comprising display disabling means for set- 
ting a flag to disable said viewer's ability to prevent 
the display of said multimedia information. 

30 

11 . An information processing method tor creating mul- 
timedia information to accompany video informa- 
tion transmitted for display to a viewer, said infor- 
mation processing method comprising: 

35 

creating said multimedia information; 
identifying video information with which said 
rriultimedia infonmation is to be synchronized; 
and 

40 setting the timing for displaying one or more 

scenes of said multimedia information so as to 
cause said display to be synchronized with the 
displaying of said video information. 

45 12. An information processing method as in claim 11, 
wherein said multimedia information conforms to 
the MHEG standard. 

13. An information processing method as in claim 11. 
so wherein said multimedia information conforms to 

the ATVEF standard. 

14. An information processing method as in claim 11. 
wherein said setting comprises setting said timing 

55 with respect to the relative time for the display of 
said video information. 

15. An information processing method as in claim 11, 
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further comprising: 

displaying an axis corresponding to the time tor 
the display of said video information; and 
displaying, relative to said axis, symbols corre- 
sponding to the timing for displaying said one 
or more scenes. 

16. An information processing method as in claim 11, 
further comprising setting a flag for said multimedia 
information for deactivating the synchronized dis- 
play, with said video information, of a scene of said 
multimedia information. 

17. An information processing method as in claim 11, 
further comprising setting a flag to prevent the dis- 
play of a scene of said multimedia information 
whose playback is synchronized to begin, with re- 
spect to said video information, at a point in said 
video information which occurs before the point at 
which said viewer begins playing back said video 
information. 

18. An information processing apparatus as in claim 1, 
further comprising organizing and managing said 
multimedia information in a hierarchy of contents, 
scenarios, and scenes. 

1 9. An information processing apparatus as in claim 1 1 . 
further comprising setting the rate at which a plural- 
ity of said scenes are transmitted to said viewer 

20. An information processing apparatus as in claim 11 , 
further comprising display disabling means for set- 
ting a flag to disable said viewer's ability to prevent 
the display of said multimedia information. 

21 . A computer-readable medium having computer-ex- 
ecutable software code stored thereon for creating 
multimedia information to accompany video infor- 
mation transmitted for display to a viewer, said soft- 
ware code comprising: 

creating code for creating said multimedia in- 
formation; 

identifying code for identifying video informa- 
tion with which said multimedia information is 
to be synchronized ; and 
setting code for setting the timing for displaying 
one or more scenes of said multimedia infor- 
mation so as to cause said displaying to be syn- 
chronized with the display of said video infor- 
mation. 

22. A computer- readable medium as in claim 21, 
wherein said multimedia information conforms to 
the MHEG standard. 
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23. A computer-readable medium as in claim 21, 
wherein said multimedia information conforms to 
the ATVEF standard. 

5 24. A computer- readable medium as in claim 21. 
wherein said setting code comprises code for set- 
ting said timing with respect to the relative time for 
the display of said video information. 

10 25. A computer-readable medium as in claim 21. 
wherein said software code further comprises: 

time-axis display code for displaying an axis 
corresponding to the time for the display of said 
IS video information; and 

display control code for displaying, relative to 
said axis, symbols corresponding to the timing 
for displaying said one or more scenes. 

20 26. A computer-readable medium as in claim 21. 
wherein said software code further comprises de- 
activation code for deactivating the synchronized 
display, with said video information, of a scene of 
said multimedia information. 

25 

27. A computer-readable medium as in, claim 21, 
wherein said software code further comprises mid- 
processing setting code for setting a flag to prevent 
the display of a scene of said multimedia informa- 

30 lion whose playback is synchronized to begin, with 
respect to said video information, at a point in said 
video information which occurs before the point at 
which said viewer begins playing back said video 
information. 

35 

28. A computer-readable medium as in claim 21, 
wherein said software code further comprises man- 
agement code for organizing and managing said 
multimedia information in a hierarchy of contents, 

40 scenarios and scenes. 

29. A computer-readable medium as in claim 21, 
wherein said software code further comprises 
transmission rate setting code for setting the rate at 

45 which a plurality of said scenes are transmitted to 
said viewer. 

30. A computer-readable medium as in claim 21, 
wherein said software code further comprise dis- 

so play disabling code for setting a flag to disable said 
viewer's ability to prevent the display of said multi- 
media information. 

31 . An apparatus for synchronizing the display of a vid- 
55 eo with multimedia information corresponding to 

said video, said apparatus comprising: 

means for specifying the duration for displaying 
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said video; 

means \o\ specifying the period for displaying 
each scene of a plurality ot scenes of said mul- 
timedia information with respect to said dura- 
tion; 

means for displaying on said computer screen 
a plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
spect to said duration, of the period corre- 
sponding to said scene. 

32. An apparatus as In claim 31, further comprising 
means for displaying on said computer screen, con- 
currently with said symbols, a second symbol rep- 
resenting the duration for displaying said video. 

33. An apparatus as in claim 31 . wherein said multime- 
dia information conforms to the fVlHEG standard. 

34. An apparatus as in claim 31 . wherein said multime- 
dia information conforms to the ATVEF standard. 

35. An apparatus for synchronizing the broadcasting of 
a video with multimedia information broadcasted 
with said video, said apparatus comprising: 

means for specifying the duration for broad- 
casting said video; 

means for specifying the period for broadcast- 
ing each scene of a plurality of scenes of said 
multimedia information with respect to said du- 
ration; 

means for displaying on said computer screen 
a plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
spect to said duration, of the period corre- 
sponding to said scene. 

36. An apparatus as in claim 35, further comprising 
means for displaying on said computer screen, con- 
currently with said symbols, a second symbol rep- 
resenting the broadcasting duration of said video. 

37. An apparatus as in claim 34, wherein said multime- 
dia information conforms to the MHEG standard. 

38. An apparatus as In claim 34, wherein said multime- 
dia information conforms to the ATVEF standard. 

39. A computer-readable medium having computer-ex- 
ecutable software code stored thereon for synchro- 
nizing the display of a video with multimedia infor- 
mation corresponding to said video, said software 
code comprising: 

code for specifying the duration for displaying 
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said video; 

code for specifying the period for displaying 
each scene of a plurality of scenes of said mul- 
timedia information with respect to said dura- 
tion; 

code for displaying on said computer screen a 
plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
spect to said duration, of the period corre- 
sponding to said scene. 

40. A computer-readable medium as in claim 39, 
wherein said software code further comprises code 
for displaying on said computer screen, concurrent- 
ly with said symbols, a second symbol representing 
the duration for displaying said video. 

41. A computer-readable medium as in claim 39, 
wherein said multimedia information conforms to 
the MHEG standard. 

42. A computer-readable medium as in claim 39, 
wherein said multimedia Information conforms to 
the ATVEF standard. 

43. A computer-readable medium having computer-ex- 
ecutable software code stored thereon for synchro- 
nizing the broadcasting of a video with multimedia 
information broadcasted with said video, said soft- 
ware code comprising: 

code for specifying the duration for broadcast- 
ing said video; 

code for specifying the period for broadcasting 
each scene of a plurality of scenes of said mul- 
timedia information with respect to said dura- 
tion; 

code for displaying on said computer screen a 
plurality of symbols, each of said symbols cor- 
responding to one of said scenes and repre- 
senting the relative time and position, with re- 
spect to said duration, of the period corre- 
sponding to said scene. 

44. A computer-readable medium as in claim 43, 
wherein said software code further comprises code 
for displaying on said computer screen, concurrent- 
ly with said symbols, a second symbol representing 
the broadcasting duration of said video. 

45. A computer-readable medium as in claim 43, 
wherein said multimedia information conforms to 
the MHEG standard. 

46. A computer-readable medium as In claim 43, 
wherein said multimedia information conforms to 
the ATVEF standard. 



9 



EP 0 942 595 A2 




10 



EP 0 942 595 A2 



FIG. 2 



C 



11 


12 

\ 


13 

i 


CPU 




ROM 




RAM 





HARD 




DISPLAY 


DISK 




APPARATUS 


\ 


\ 


14 


15 



KEYBOARD 



16 



MOUSE 



"T 

17 



19 

L_ 

VOICE 

PLAYBACK 

APPARATUS 



COMMUNI- 
CATION 
SECTION 

r 

18 



11 



EP 0 942 595 A2 



FIG. 3 



31 

L_ 

VIDEO AND MHEG 
DISPLAY CONTROL 
SECTION 



30 

J_ 

MHEG 

PLAYBACK 

SECTION 



29 

L 

VIDEO 

PLAYBACK 

SECTION 



27 
_i_ 



TIME 

CONTROL 
SECTION 



21 

_L 



CONTENTS 

CREATION 

SECTION 



24 



CONTENTS 

MANAGEMENT 

SECTION 



28 



INTERACTIVE 

CONTROL 

SECTION 



22 

J_ 



SCENARIO 
CREATION 
SECTION 



25 



SCENARIO 

MANAGEMENT 

SECTION 



23 



SCENE 

CREATION 

SECTION 



26 



SCENE 

MANAGEMENT 
SECTION 



12 



EP 0 942 595 A2 



FIG. 4 



SCENE 1 



r 



OBJECT 



SCENE 2 

\ 



SCENE 3 

\ 



IMAGE 



TEXT 



OPERATION 
BUTTON 



SCENARIO SCI 



FIG. 5 



CI 

\ 

CONTENTS 



SCI 

L_ 

SCENARIO 



SC2 



SCENARIO 



SCENARIO 

H 

SC3 



SCENE 1 



SCENE 2 



SCENE 3 



SCENE 4 



SCENE 5 



13 



EP 0 942 595 A2 



CO 

O 




. GO 



CO 



to 

CO 



3| 

o <: 



14 



EP 0 942 595 A2 



FIG. 7 



56 

i 

VIDEO DISPLAY 

CONTROL 

SECTION 



52 
J_ 



TIME 

CONTROL 
SECTION 



55 

L_ 

MHEG 

PLAYBACK 

SECTION 



54 

L. 

VIDEO 

PLAYBACK 

SECTION 



53 



INTERACTIVE 

CONTROL 

SECTION 



51 



DIRECTORY 

MANAGEMENT 

SECTION 



15 



EP 0 942 595 A2 



FIG. 8 



C SYNCHRONIZATION SETTING ^ 
CONFIRMATION PROCESS START J 





r 


DISPLAY TIME AXIS OF VIDEO 
TO BE SYNCHRONIZED 


> 




READ RELATIVE TIME AND 
DURATION 




f 


DISPLAY RELATIVE TIME OF 
SCENARIO AND DURATION ALONG 
TIME AXIS OF VIDEO SIGNAL 




f 



S1 



( END ) 



16 



EP 0 942 595 A2 



FIG. 9 



VIDEO 
CONTENTS 



VIDEO 
START 



VIDEO 
END 



10 



20 



30 (MIN.) 



SCENARIO 
SC1 



PLAYBACK START TIME 
AT 0 MIN. 

PLAYBACK DURATION 
10 MIN. 



SCENARIO 
SC2 



SENDING START TIME 
AT -5 MIN. 
SENDING DURATION 
20 MIN. 



PLAYBACK START TIME 
AT 20 MIN. 
PLAYBACK DURATION 
10 MIN. 

SENDING START TIME 
AT 15 MIN. 
SENDING DURATION 
20 MIN. 



17 



EP 0 942 595 A2 



FIG. 10 

C INTERACTIVE SETTING CONFIRMATION^ 
V PROCESS START ) 

S21 



DISPLAY TIME AXIS OF VIDEO SIGNAL 
TO BE SYNCHRONIZED 



I 



READ PLAYBACK START TIME OF 

Fl RST SCENE AND PLAYBACK DURATION 



S22 



DISPLAY SYMBOL OF ONE SCENE FROM AMONG THE 
SYMBOL OF THE FIRST SCENE AND THE SYMBOL OF 
THE SCENE WHICH CAN BE SELECTED THERE 



S23 



■^SYMBOL OF SELECTABLE SCENE SELECTED?) S24 





PULL-DOWN DISPLAY SYMBOLS OF 
ALL SELECTABLE SCENES 






r 


ml 





' ONE OF THE SYMBOLS OF ALL SELEC- 



, ) 

|YES 



TABLE SCENES HAS BEEN ^'^-^^^'^ ' ^26 



DISPLAY SYMBOL OF SELECTED SCENE AND SYMBOL 
OF SCENE WHICH CAN BE SELECTED THERE IN AN 
ORDER LOWER THAN SYMBOL OF FIRST SCENE 



S27 



mJ SYMBOL OF SELECTABLE SCENE HAS \ 
\ BEEN SELECTED? /^^^ 

S29 



PULL-DOWN DISPLAY SYMBOLS OF ALL 
SELECTABLE SCENES 



NO 



/ ONE OF THE SYMBOLS OF ALL SELEC- Xcon 
\ TABLE SCENES HAS" BEEN SELECTED? / 



DISPLAY SYMBOL OF SELECTED SCENE AND SYMBOL 
OF SCENE WHICH CAN BE SELECTED THERE IN AN 
ORDER LOWER THAN SYMBOL OF HIGH- ORDER SCENE 



S31 



C END ) 



18 



EP 0 942 595 A2 



FIG. 11 



SCENE 1 



SCENE 2 




SCENE 6 










SCENE 3 






SCENE 7 






SCENE 4 






SCENE 8 



SCENE 5 



19 



EP 0 942 595 A2 



FIG. 12 



VIDEO 
START 



VIDEO 



10 



20 



VIDEO 
END 



30 (MIN.) 



SYMBLOL-- 



SCENE 1 


SCENE 2> 


I 


SCENE 1 


SCENE 2 




SCENE 3 




SCENE 4 




SCENE 5 


J 


i 


SCENE. 1 




SCENE 2 


SCENE 6^ 


i 


SCENE 1 




SCENE 2 


SCENE 6 




SCENE 7 




SCENE 8 




I 


SCENE 1 




SCENE 2 




SCENE 6 





20 



EP 0 942 595 A2 
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